<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
<title>Papyros: Lightweight master-slave parallel processing in Python</title>
<style type="text/css">

/* nuxeo doc css */
/* $Id: nuxeo_doc.css 30369 2005-12-07 09:14:45Z madarche $ */

body {
  font: 90% 'Lucida Grande', Verdana, Geneva, Lucida, Arial, Helvetica, sans-serif;
  background: #ffffff;
  color: black;
  margin: 2em;
  padding: 2em;
}

a[href] {
  color: #436976;
  background-color: transparent;
}

a.toc-backref {
  text-decoration: none;
}

h1 a[href] {
  text-decoration: none;
  color: #fcb100;
  background-color: transparent;
}

a.strong {
  font-weight: bold;
}

img {
  margin: 0;
  border: 0;
}

p {
  margin: 0.5em 0 1em 0;
  line-height: 1.5em;
}
p a {
  text-decoration: underline;
}
p a:visited {
  color: purple;
  background-color: transparent;
}
p a:active {
  color: red;
  background-color: transparent;
}
a:hover {
  text-decoration: none;
}
p img {
  border: 0;
  margin: 0;
}

h1, h2, h3, h4, h5, h6 {
  color: #003a6b;
  background-color: transparent;
  font: 100% 'Lucida Grande', Verdana, Geneva, Lucida, Arial, Helvetica, sans-serif;
  margin: 0;
  padding-top: 0.5em;
}

h1 {
  font-size: 160%;
  margin-bottom: 0.5em;
  border-bottom: 1px solid #fcb100;
}
h2 {
  font-size: 140%;
  margin-bottom: 0.5em;
  border-bottom: 1px solid #aaa;
}
h3 {
  font-size: 130%;
  margin-bottom: 0.5em;
}
h4 {
  font-size: 110%;
  font-weight: bold;
}
h5 {
  font-size: 100%;
  font-weight: bold;
}
h6 {
  font-size: 80%;
  font-weight: bold;
}

ul a, ol a {
  text-decoration: underline;
}

dt {
  font-weight: bold;
}
dt a {
  text-decoration: none;
}

dd {
  line-height: 1.5em;
  margin-bottom: 1em;
}

legend {
  background: #ffffff;
  padding: 0.5em;
}

form {
  margin: 0;
}


dl.form {
  margin: 0;
  padding: 1em;
}

dl.form dt {
  width: 30%;
  float: left;
  margin: 0;
  padding: 0 0.5em 0.5em 0;
  text-align: right;
}

input {
  font: 100% 'Lucida Grande', Verdana, Geneva, Lucida, Arial, Helvetica, sans-serif;
  color: black;
  background-color: white;
  vertical-align: middle;
}

abbr, acronym, .explain {
  color: black;
  background-color: transparent;
}

q, blockquote {
}

code, pre {
  font-family: monospace;
  font-size: 1.2em;
  display: block;
  padding: 10px;
  border: 1px solid #838183;
  background-color: #eee;
  color: #000;
  overflow: auto;
  margin: 0.5em 1em;
}

tt.docutils {
  background-color: #eeeeee;
}

</style>
</head>
<body>
<div class="document" id="papyros-lightweight-master-slave-parallel-processing-in-python">
<h1 class="title">Papyros: Lightweight master-slave parallel processing in Python</h1>

<p><strong>Papyros</strong> is a small platform independent parallel processing Python package, providing a master-slave model. Clients can submit jobs to a master object which is monitored by one or more slave objects that do the real work and return the processed jobs back to the master. Two main implementations are currently provided, one using multiple threads and one multiple processes in one or more hosts (through <a class="reference" href="http://pyro.sourceforge.net/">Pyro</a>). The primary design goal is simplicity: a user can replace a time consuming loop in a single-threaded single-process program with the equivalent parallel version in a few lines of code, with minimal boilerplate overhead.</p>
<p>Bug reports, feature requests and patches are <a class="reference" href="http://code.google.com/p/papyros/issues/list">welcome</a>. No promises though; I can't tell how often or for how long I will actively maintain the project, so people interested in participating may <a class="reference" href="mailto:george.sakkis&#64;gmail.com">contact me</a> as well.</p>
<p><em>George Sakkis</em></p>
<div class="section" id="installation">
<h1>Installation</h1>
<p>Installation uses the standard distutils command; just run from the distibution's directory:</p>
<pre class="literal-block">
python setup.py install
</pre>
<p>Papyros has been (lightly) tested on Windows XP (Python 2.5, Pyro 3.7) and Linux (Python 2.4/2.5, Pyro 3.7). Pyro is required only for the distributed implementation; the single-process implementations don't need it.</p>
</div>
<div class="section" id="usage">
<h1>Usage</h1>
<p>The typical steps for using Papyros are outlined below. For more details consult the <a class="reference" href="docs/index.html">documentation</a>.</p>
<div class="section" id="create-the-job-type-s">
<h2>1. Create the job type(s)</h2>
<p>Each type of task to be processed concurrently must extend the callable <tt class="docutils literal"><span class="pre">papyros.Job</span></tt> class and override its <tt class="docutils literal"><span class="pre">__call__</span></tt> method. Any number of positional and named arguments are allowed. A prime factorization job can simply be:</p>
<pre class="literal-block">
import papyros

class Factorization(papyros.Job):
    '''A job for factorizing a single number.'''

    def __call__(self, num):
        # &lt;-- find the prime factors here --&gt; #
        return primes
</pre>
</div>
<div class="section" id="create-a-master-object">
<h2>2. Create a <tt class="docutils literal"><span class="pre">Master</span></tt> object</h2>
<p>The master object implements the <tt class="docutils literal"><span class="pre">papyros.Master</span></tt> interface. Currently two main implementations are provided, a multithreaded and a distributed one <a class="footnote-reference" href="#id3" id="id1">[1]</a>:</p>
<ul>
<li><p class="first"><strong>Multithreaded mode</strong></p>
<pre class="literal-block">
from papyros.multithreaded import MultiThreadedMaster
# create a master with 3 slave threads
master = MultiThreadedMaster(3)
</pre>
</li>
<li><p class="first"><strong>Distributed mode</strong></p>
<p>There are four steps to use Papyros in distributed mode. These steps don't have to be taken on the same host; in fact each of them may be run on a separate machine. This mode requires <a class="reference" href="http://pyro.sourceforge.net/">Pyro</a> to be installed on all participating hosts <a class="footnote-reference" href="#id4" id="id2">[2]</a>.</p>
<ol class="arabic">
<li><p class="first">Start the Pyro name server:</p>
<pre class="literal-block">
pyro-ns
</pre>
</li>
<li><p class="first">Start the master process:</p>
<pre class="literal-block">
python -m papyros.distributed [--logging] [--group=process_group] [--config=Pyro_config_file]
</pre>
<p><tt class="docutils literal"><span class="pre">process_group</span></tt> is an optional name identifying the master-slave group (by default <tt class="docutils literal"><span class="pre">papyros.distributed.DEFAULT_GROUP</span></tt>)</p>
</li>
<li><p class="first">Start one or (typically) more slave processes on one or more machines:</p>
<pre class="literal-block">
python -m papyros.distributed --slaves=3 [--logging] [--group=process_group] [--config=Pyro_config_file]
</pre>
<p>This starts 3 slaves processes on the machine it was issued. Repeat this for every machine in the group.</p>
</li>
<li><p class="first">Create the master object in the client:</p>
<pre class="literal-block">
from papyros.distributed import DistributedMaster
master = DistributedMaster(process_group)
</pre>
<p>As in the previous steps, the <tt class="docutils literal"><span class="pre">process_group</span></tt> is optional. If specified, it must be the same in all steps.</p>
</li>
</ol>
<p>Steps (2) and (3) are actually independent. The slave processes may be started before the master and they will connect automatically to their master once it starts.</p>
</li>
</ul>
</div>
<div class="section" id="submit-a-number-of-jobs-to-the-master">
<h2>3. Submit a number of jobs to the master</h2>
<p>Given a master object, the client can add any number of jobs to be processed. Each job is created by passing the arguments that are expected by its <tt class="docutils literal"><span class="pre">__call__</span></tt>:</p>
<pre class="literal-block">
# compute the primes factors of 10 random numbers
import random
for _ in xrange(10):
    master.addJob(Factorization(random.randrange(1e6,1e7)))
</pre>
<p>Of course it's not mandatory to add all the jobs upfront but that's a common approach.</p>
</div>
<div class="section" id="receive-the-processed-jobs-from-the-master">
<h2>4. Receive the processed jobs from the master</h2>
<p>Once one or more jobs are submitted, the client can ask from the master the next available processed job by calling <tt class="docutils literal"><span class="pre">popProcessedJob</span></tt>:</p>
<pre class="literal-block">
# blocks until a job has finished and returns it
job1 = master.popProcessedJob()
# non-blocking call; returns a processed job if there is one available immmediately, or None otherwise
job2 = master.popProcessedJob(timeout=0)
# blocks no more than 2.5 seconds; returns None if there's no available job after the deadline
job3 = master.popProcessedJob(timeout=2.5)
</pre>
<p><tt class="docutils literal"><span class="pre">popProcessedJob</span></tt> returns either None or one of the previously submitted jobs. In the latter case, the processed job encapsulates the result of the job or the raised exception:</p>
<pre class="literal-block">
try: print 'Job %s returned %r' % job.result
except Exception, ex:
    print 'Job %s raised an exception: ' % ex
    # the original traceback printout is also available through the traceback attribute
    print 'Original traceback\n%s' % ex.traceback
</pre>
<p>A handy idiom to loop over the processed jobs is by using the <tt class="docutils literal"><span class="pre">iter</span></tt> builtin in its less known form <tt class="docutils literal"><span class="pre">iter(callable,</span> <span class="pre">sentinel)</span></tt>:</p>
<pre class="literal-block">
# loop over all pending jobs
for job in iter(master.popProcessedJob, None):
    print job.result
# loop over pending jobs as long each job is available within 2 seconds from the previous one
for job in iter(lambda: master.popProcessedJob(timeout=2), None):
    print job.result
</pre>
<p>From the last example it can be inferred that if there are 30 pending jobs, the whole loop will take at most 1 minute; for 60 jobs it would be at most 2 minutes, etc. For cases where there has to be an upper limit for the loop independently of the number of jobs, the <tt class="docutils literal"><span class="pre">processedJobs</span></tt> method comes in handy:</p>
<pre class="literal-block">
# returns a list of jobs available within 10 seconds
jobs = master.processedJobs(timeout=10)
print '%s jobs finished' % len(jobs)
</pre>
</div>
</div>
<div class="section" id="examples">
<h1>Examples</h1>
<p>The <tt class="docutils literal"><span class="pre">examples/demo.py</span></tt> script illustrates the Papyros API and its three implementations (singlethreaded, multithreaded, distributed). There are two sample job types, a cpu-intensive (<tt class="docutils literal"><span class="pre">PrimeFactors</span></tt>) and a non-cpu-intensive (<tt class="docutils literal"><span class="pre">SlowSqrt</span></tt>). The demo also show the <tt class="docutils literal"><span class="pre">cancelAllJobs</span></tt> method that discards all currently unassigned jobs (it has no effect on jobs already assigned to a slave).</p>
<p>To run the demo, just give <tt class="docutils literal"><span class="pre">python</span> <span class="pre">examples/demo.py</span></tt> and answer the questions when prompted. Note for that for the distributed version, both the master and the slave processes should be able to import <tt class="docutils literal"><span class="pre">papyros</span></tt> and the <tt class="docutils literal"><span class="pre">jobs.py</span></tt> module under the <tt class="docutils literal"><span class="pre">examples</span></tt> directory.</p>
<div class="section" id="notes">
<h2>Notes</h2>
<table class="docutils footnote" frame="void" id="id3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>There is also a third, non-concurrent version, mainly for the sake of completeness and as a fallback in systems where the concurrent versions are not available.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id4" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>The module(s) required by the job types to be processed must also be importable from all hosts. In a future version this restriction may be raised by enabling the PYRO_MOBILE_CODE feature.</td></tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="footer">
<hr class="footer" />
Generated on: 2007-06-11.
Generated by <a class="reference" href="http://docutils.sourceforge.net/">Docutils</a> from <a class="reference" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> source.

</div>
</body>
</html>
